The Accuracy of Alternative Extrapolation Models: Analysis of a Forecasting Competition Through Open Peer Review

In 1982, the Journal of Forecasting published the results of a forecasting competition organized by Spyros Makridakis (Makridakis et al., 1982). In this, the ex ante forecast errors of 21 methods were compared for forecasts of a variety of economic time series, generally using 1001 time series. Only extrapolative methods were used, as no data were available on causal variables. The accuracies of methods were compared using a variety of accuracy measures for different types of data and for varying forecast horizons. The original paper did not contain much interpretation or discussion. Partly this was by design, to be unbiased in the presentation. A more important factor, however, was the difficulty in gaining consensus on interpretation and presentation among the diverse group of authors, many of whom have a vested interest in certain methods. In the belief that this study was of major importance, we decided to obtain a more complete discussion of the results. We do not believe that “the data speak for themselves.” The Ground Rules In seeking peer review of the Makridakis competition, we drew heavily upon the procedures used by Behavioral and Brain Sciences, a journal that has been one of the pioneers for open peer review (Harnad, 1979). One objective was to provide a forum for discussion by experts who were likely to have different perspectives. We invited 14 outside experts to write commentaries on the Makridakis competition. Of these, eight agreed and seven completed their papers. The commentators are all from different organizations. Three are practitioners and four are academicians. We asked these commentators to address any aspect of the original paper. They were given approximately five months to write their commentary. We reviewed each commentary and made suggestions for change (sometimes with more than one round of revisions). Later, the commentators were provided with papers by the other commentators and were given a further opportunity for revisions. Finally, the commentators and authors were provided with edited versions of all papers and were given an opportunity to clarify their own papers and to suggest clarifications in other papers. A second objective was to obtain the viewpoints of the original authors speaking freely without any need for agreement from their co-authors. We also sent the commentaries to each of the nine original authors. We received replies from seven of the nine authors. Received March 1983 Some of the authors of the Makridakis competition were associated with the Journal of Forecasting as editors or associate editors. To provide an independent assessment, these authors removed themselves from editorial decisions.